Class Readings, Assignments, Syllabus Topics
Reading, Lab Exercises, SemProjects
- Readings:
- For today:
- For next class: LeCun Deep Learning 2015 Review
- Next is:
- Laboratory Exercises:
- LE posted
- LE6 is due this coming Thursday April 13th
- Office Hours: (Class Canvas Calendar for Zoom Link)
- Wednesdays @ 4:00 PM to 5:00 PM
- Saturdays @ 3:00 PM to 4:00 PM
- Office Hours are on Zoom, and recorded
- Semester Projects
- Office Hours for SemProjs: Mondays at 4pm on Zoom
- DSCI 453 Students Biweekly Updates Due
- DSCI 453 Students
- All DSCI 353/353M/453, E1453/2453 Students:
- Peer Grading of Report Out is Due Friday
- Exams
- Final: Tuesday May 11th 2021, 12:00 PM to 3 PM
The recipients of the 2019 Turing Award of ACM
Geoffrey Hinton, Yann Lecun and Yoshua Bengio
- Geoffrey
Hinton
- Backpropagation: In a 1986 paper, “Learning
Internal Representations by Error Propagation,” co-authored with David
Rumelhart and Ronald Williams, Hinton demonstrated that the
backpropagation algorithm allowed neural nets to discover their own
internal representations of data, making it possible to use neural nets
to solve problems that had previously been thought to be beyond their
reach. The backpropagation algorithm is standard in most neural networks
today.
- Boltzmann Machines: In 1983, with Terrence
Sejnowski, Hinton invented Boltzmann Machines, one of the first neural
networks capable of learning internal representations in neurons that
were not part of the input or output.
- Improvements to convolutional neural networks: In
2012, with his students, Alex Krizhevsky and Ilya Sutskever, Hinton
improved convolutional neural networks using rectified linear neurons
and dropout regularization. In the prominent ImageNet competition,
Hinton and his students almost halved the error rate for object
recognition and reshaped the computer vision field.
- Yann Lecun
- Convolutional neural networks: In the 1980s, LeCun
developed convolutional neural networks, a foundational principle in the
field, which, among other advantages, have been essential in making deep
learning more efficient. In the late 1980s, while working at the
University of Toronto and Bell Labs, LeCun was the first to train a
convolutional neural network system on images of handwritten digits.
Today, convolutional neural networks are an industry standard in
computer vision, as well as in speech recognition, speech synthesis,
image synthesis, and natural language processing. They are used in a
wide variety of applications, including autonomous driving, medical
image analysis, voice-activated assistants, and information
filtering.
- Improving backpropagation algorithms: LeCun
proposed an early version of the backpropagation algorithm (backprop),
and gave a clean derivation of it based on variational principles. His
work to speed up backpropagation algorithms included describing two
simple methods to accelerate learning time.
- Broadening the vision of neural networks: LeCun is
also credited with developing a broader vision for neural networks as a
computational model for a wide range of tasks, introducing in early work
a number of concepts now fundamental in AI. For example, in the context
of recognizing images, he studied how hierarchical feature
representation can be learned in neural networks—a concept that is now
routinely used in many recognition tasks. Together with Léon Bottou, he
proposed the idea, used in every modern deep learning software, that
learning systems can be built as complex networks of modules where
backpropagation is performed through automatic differentiation. They
also proposed deep learning architectures that can manipulate structured
data, such as graphs.
- Yoshua
Bengio
- Probabilistic models of sequences: In the 1990s,
Bengio combined neural networks with probabilistic models of sequences,
such as hidden Markov models. These ideas were incorporated into a
system used by AT&T/NCR for reading handwritten checks, were
considered a pinnacle of neural network research in the 1990s, and
modern deep learning speech recognition systems are extending these
concepts.
- High-dimensional word embeddings and attention: In
2000, Bengio authored the landmark paper, “A Neural Probabilistic
Language Model,” that introduced high-dimension word embeddings as a
representation of word meaning. Bengio’s insights had a huge and lasting
impact on natural language processing tasks including language
translation, question answering, and visual question answering. His
group also introduced a form of attention mechanism which led to
breakthroughs in machine translation and form a key component of
sequential processing with deep learning.
- Generative adversarial networks: Since 2010,
Bengio’s papers on generative deep learning, in particular the
Generative Adversarial Networks (GANs) developed with Ian Goodfellow,
have spawned a revolution in computer vision and computer graphics. In
one fascinating application of this work, computers can actually create
original images, reminiscent of the creativity that is considered a
hallmark of human intelligence.
Hinton and LeCun’s Turing Award Lecture recordings
Types of Neural Networks
There are many different types of neural networks
- which function on the same principles
- as the nervous system in the human body.
As Howard Rheingold said,
“The neural network is this kind of technology that is not an algorithm,
it is a network that has weights on it,
and you can adjust the weights so that it learns.
You teach it through trials.”
What are Artificial Neural Networks?
Artificial neural networks are
- a variety of deep learning technology
- which comes under the broad domain
- of Artificial Intelligence.
Deep learning is a branch of Machine Learning
- which uses different types of neural networks.
These algorithms are inspired by the way our brain functions
- and therefore many experts believe they are our best shot
- to moving towards real AI (Artificial Intelligence).
Deep learning is becoming especially exciting now
- as we have more amounts of data
- and larger neural networks to work with.
Moreover, the performance of neural networks
- improves as they grow bigger and work with more and more data,
- unlike other Machine Learning algorithms
- which can reach a plateau after a point.
Figure 1. A mostly complete chart of Neural Networks
How do Neural Networks work?
Each successive layer then receives input
- from the layer before it
- and then passes on its output to the layer after it.
- The last layer processes the final output.
Nodes make up each tier.
- The nodes are highly interconnected
- with the nodes in the tier before and after.
- Each node in the neural network has its own sphere of knowledge,
- including rules that it was programmed with
- and rules it has learnt by itself.
The key to the efficacy of neural networks
- is they are extremely adaptive and learn very quickly.
- Each node weighs the importance of the input
- it receives from the nodes before it.
- The inputs that contribute the most towards the right output
- are given the highest weight.
What are the Different Types of Neural Networks?
There are many types of artificial neural networks, each with their
unique strengths.
This Hinton video discusses the
- different types of neural networks
- and their applications in detail.
Neural Networks for Machine
Learning, Class 2a, Geoffrey Hinton
Figure 2. Geoffrey Hinton on Types of Neural Networks
Geoffrey Hinton
- He has developed many of the most important advances in machine
learning.
These papers are in your 3-readings / 2-articles
- Backpropagation [@rumelhartLearningRepresentationsBackpropagating1986]
- t-SNE for data visualization [@maatenVisualizingDataUsing2008]
- Deep NNs for ImageNet Image Classification [@krizhevskyImageNetClassificationDeep2012a,@krizhevskyImageNetClassificationDeep2017]
- Deep Learning [@lecunDeepLearning2015]
- His Neural
Networks for Machine Learning class videos playlist
Important types of neural networks and their applications
Feedforward
Neural Network – Artificial Neuron
This is one of the simplest types of artificial neural networks.
- In a feedforward neural network,
- the data passes through the different input nodes
- till it reaches the output node.
In other words, data moves in only one direction
- from the first tier onwards
- until it reaches the output node.
- This is also known as a front propagated wave
- which is usually achieved
- by using a classifying activation function.
Unlike in more complex types of neural networks,
- there is no backpropagation
- and data moves in one direction only.
- A feedforward neural network
- may have a single layer
- or it may have hidden layers.
In a feedforward neural network,
- the sum of the products of the inputs and their weights are
calculated.
- This is then fed to the output.
Here is an example of a single layer feedforward neural network.
Figure 3. Feedforward Neural Network – Artificial Neuron
Feedforward neural networks are used in technologies like
- Face recognition and computer vision.
- This is because the target classes in these applications
A simple feedforward neural network is equipped to deal with data
- which contains a lot of noise.
- Feedforward neural networks are also relatively simple to
maintain.
Radial Basis Function Neural Network
A radial basis function
- considers the distance of any point relative to the centre.
Such neural networks have two layers.
- In the inner layer,
- the features are combined with the radial basis function.
Then the output of these features is taken into account
- when calculating the same output in the next time-step.
Here is a diagram which represents a radial basis function neural
network.

knitr::include_graphics[width = 0.8\textwidth]("./figs/TypeNN-4-RBFNN.gif")
Figure 4. Radial Basis Function Neural Network
The radial basis function neural network is applied extensively
- in power restoration systems.
- In recent decades, power systems have become bigger and more
complex.
- This increases the risk of a blackout.
This neural network is used in the power restoration systems
- in order to restore power in the shortest possible time.
Multilayer Perceptron
A multilayer perceptron
- uses a nonlinear activation function
- (mainly hyperbolic tangent or logistic function).
Here’s what a multilayer perceptron looks like.
Figure 5. Multilayer Perceptron
This type of neural network is applied extensively
- in speech recognition
- and machine translation technologies.
Convolutional Neural Network
A CNN contains one or more than one convolutional layers.
- These layers can either be
- completely interconnected
- or pooled.
Before passing the result to the next layer,
- the convolutional layer
- uses a convolutional operation on the input.
- Due to this convolutional operation,
- the network can be much deeper
- but with much fewer parameters.
Due to this ability, convolutional neural networks show very
effective results
- in image and video recognition,
- natural language processing,
- and recommender systems.
Convolutional neural networks also show great results
- in semantic parsing and paraphrase detection.
They are also applied in
- signal processing
- and image classification.
CNNs are also being used in image analysis and recognition in
agriculture
- where weather features are extracted from satellites
- to predict the growth and yield of a piece of land.
Here’s an image of what a Convolutional Neural Network looks
like.
Figure 6. Convolutional Neural Network
Figure 6b. Our ConvNet used for Image Machine Learning of PV Cell
Degradation